Skip to content

Speed up Cursor.fetchall by draining batches directly#196

Open
TheDistributor wants to merge 1 commit into
masterfrom
mgallwey/accelerate-cursor
Open

Speed up Cursor.fetchall by draining batches directly#196
TheDistributor wants to merge 1 commit into
masterfrom
mgallwey/accelerate-cursor

Conversation

@TheDistributor

@TheDistributor TheDistributor commented Jun 29, 2026

Copy link
Copy Markdown
  • fetchall: extend the result list with each in-memory batch in one call and
    request the next batch from the session, instead of looping through
    fetchone() for every row
  • fetchone: inline the closed-cursor / closed-connection checks to avoid a
    function call on every row

Accelerate data retrieval pathways in the Cursor class by optimizing row
iteration and reducing interpreter overhead on high-frequency method loops.

Detailed changes:
- fetchone: Inline the `_check_closed()` state verification directly into the
  method body. This removes function-call stack overhead for every individual
  row retrieved.
- fetchall: Refactor the loop to slice and drain the current in-memory result
  batch in bulk using `list.extend()` instead of sequentially invoking
  `fetchone()` row by row.
- fetchall: Directly request the next network payload batch via
  `fetch_result_set_next` once the in-memory collection is exhausted, updating
  total `rownumber` tracking as a single block operation.
- fetchall: Add a defensive guard clause against unallocated result sets to
  ensure strict DB-API compatibility.
@TheDistributor TheDistributor self-assigned this Jun 29, 2026
@TheDistributor TheDistributor changed the title Optimize result fetching via block-draining and check inlining Optimize data retrieval via the cursor object Jun 29, 2026
@TheDistributor TheDistributor changed the title Optimize data retrieval via the cursor object Speed up Cursor.fetchall by draining batches directly Jun 29, 2026
@TheDistributor TheDistributor requested review from bkelly-ndb and ribram and removed request for madscientist June 30, 2026 08:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant